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ABSTRACT 


Self adaptive filters adjust their parameters to perform 
an almost optimal filtering operation without apriori know- 
ledge of the input signal statistics. Two approaches to the 
design of efficient self adaptive discrete filtering algorithms 
are considered. 

For non-recursive (FIR) adaptive filters, simplified esti- 
mations of the gradient of the performance function to be 
Minimized are considered. These algorithms result in reduced 
complexity of implementation, improved dynamic operating range 
with about the same misadjustment errors and convergence time 
as the classic LMS (Lease Means Squared) algorithm. An analy- 
Sis of the simplified gradient approach is presented and con- 
firmed experimentally for the specific example of an adaptive 
line enhancer (ALE). The results are used to compare the 
Simplified gradient approaches with each other and the LMS 
algorithm. This comparison is done uSing a new graphic pre- 
seiitation of adaptive filter operating characteristics and a 
complexity index. This comparison indicates that the simplified 
gradient estimators are superior to the LMS algorithm for 
filters of equal complexity. 

For recursive (IIR) adaptive filters a combined random 
and gradient search (RGS) algorithm is proposed, analyzed and 
tested. Since for the IIR filter, the performance surface is 
multimodal in the feedback parameters and unimodal in the 


feedforward parameters, random search is used to adjust the 





feedback parameters and gradient search to adjust the feed- 
forward parameters. Convergence to the globally optimal 
filter parameters is guaranteed for sufficiently long 
adaptation time. Convergence time estimation for the RGS 
algorithm is derived and supported by simulation results for 
the ALE example. Finally, apriori knowledge of the optimal 
filter structure is taken into account in the formulation of 
an improved version of the basic RGS algorithm. This improve- 


ment is confirmed with the ALE example. 
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i. INTRODUCTION 


1.1. BACKGROUND 
In a broad sense the term filter implies an operation 

on an input signal or collection of data in order to smooth, 
predict, or estimate a desired property hidden in the input. 
Fig. 1.1-1 presents the block diagram of a discrete time 
linear recursive digital filter. An optimal filter is one 
designed to be optimum or best with respect to a performance 
criterion that measures or expresses its effectiveness. The 
most commonly used approach to optimal filter design is the 
linear filter optimized with respect to a Minimum Mean 
Squared Error (MMSE), where the error is defined as the 
difference between the filter output and a desired signal. 

This optimal filter is usually called the Wiener filter. 
Filter realization may be for: (a) analog signals and con- 
tinuous time, (b) analog signals and discrete time, (c) di- 
gital signals and discrete time. This dissertation is 
applicable to cases (b) and (c). A basic discussion of 
discrete Wiener filters is presented by Nahi [28, Ch 5]. 
As expected the parameters of the optimal filter depend upon 
properties of the input and desired signals. For example, 
the Wiener filter solution depends upon the second order 


statistics of the input signal and the desired signal. 
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The performance surface describes the filter performance 
criterion as a function of its weights (parameters, coeffi- 
clents-a,, b> of Fig. 1.l-1). Each point of the surface 
is the value of the performance criterion with specific 
weights of the filter. The term performance function will 
be used to describe the performance criterion values as 
function of time during the adaptation process. The optimal 
filter weights are those at the global minimum point of the 
performance surface. 

In those cases where the information (input statistics) 
needed to design an optimal filter is not available, or in 
those cases where the filter is required to operate under 
statistically nonstationary input signal conditions, the 
usual optimal design approach is not applicable. In some of 
these cases, a self adaptive filter can be used to overcome 
this lack of information. The adaptive filter tries to 
adjust its parameters dynamically to variations in the sta- 
tistics of the input signal. For the weight adjustment, or 
adaptation, the adaptive filter uses an error signal. 

Ideally this error is the difference between the filter out- 
put and a desired signal. In many applications the desired 
Signal is not available per se, so that a reference signal, 
related to the desired signal in some way, is used to develop 
the error signal. Fig. 1.1-2 presents a block diagram of an 
adaptive filter with its input, output and reference signals. 
The adaptive filter thus includes a signal processing section 


which is similar to a non-adaptive filter, except that the 
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filter weights are adjustable and controlled by the second 
portion of the adaptive filter-namely the weight adaptation 
algorithm. The weight optimization algorithm typically 
estimates the gradient of the performance surface and 

adjusts the weights in the direction of steepest descent. 
For.a statistically stationary situation, after some tran- 
sient, the adaptive filter can be expected to reach a steady- 
state condition at which the parameters jitter around the 
minimum point of the performance surface. 

The generation of the reference signal is a key consi- 
deration in adaptive filter implementation. There are 
various practical methods as discussed in [1, 2, 3, 7, 22, 
24, 26, 29, 32, 37, 38, 39]. In many of these applications 
the reference signal is not identical to the signal we would 
like to have as output of the filter because if we had the 
desired output we wouldn't need the filter. In spite of 
the approximations involved, the adaptive filter is still 
able to operate and optimize the weights in many practical 
applications. 

This dissertation investigates two approaches to effi- 
cient adaptive filters. Chapter II discusses simplified 
gradient estimation methods for non-recursive filters and 
Chapter III discusses recursive filters based on a combined 


random and gradient search adaptation technique. 
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i Seer TR®™ ADAPTIVE FILTERS 

Muerte liste usiinples >: form of digital filter. 
The processing operation produces an output which in the 
linear sum of weighted delayed input samples. The impulse 
response of this filter is given by the sequence of values 
of the filter weights. Because of its relative simplicity, 
the FIR adaptive filter historically was the starting point 
for the development of adaptive filters. 

A very important property of this filter is that its 
performance surface is quadratic so we have one and only one 
minimum, i.e. it is a unimodal surface as shown by Widrow [1]. 
For a unimodal surface, a gradient minimum seeking algorithm 
Will converge to the minimum (a formal proof is presented in 
[1]), and this.property is the key to the success of the 
Least Mean Squared (LMS) algorithm, discussed later. Inter- 
est in the area of adaptive filtering started in the late 
50's and early 60's. The most successful approach is Widrow's 
LMS algorithm. Widrow in [1] presents the classic LMS algori- 
thm and summarizes most of the previous work on the subject. 
The LMS algorithm and its basic properties are presented 
later. In [3] Widrow et al introduces the concept of noise 
cancelling which uses a reference signal that is related only 
to the noise to estimate the noise portion of the input. The 


ae 


* FIR CFinite Impulse Response) and IIR (Infinite Impulse 
Response) are generally used by the signal processing community 
to denote non-recursive: and recursive. filters respectively 
and are so used in this work. It is noted though, that some 
recursive filters can have a finite impulse response. 
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output is produced by subtracting the noise estimate from 
the input signal. In [4] Widrow et al extended the analy- 
sis to non-stationary operation of the LMS algorith. In 
this study they identify two sources for misadjustment (a 
measure of the distance of the actual steady-state error 
from the optimal steady-state error) with nonstationary 
input signal. The first is due to gradient estimation 
errors (or gradient noise) which also exists with stationary 
inputs. The second cause of misadjustment with a non- 
stationary input is due to the changing statistics, and 
results ina lag in updating the filter weights after the 
optimal solution. This analysis gives some insight to the 
problem and provides basic design information. In [5] 
Widrow and McCool present a random search FIR filter and 
compare it to the LMS algorithm. Using the unimodal pro- 
perty of the FIR filter they modify the random search al- 
gorithm so that high performance function value points 
(which in regular random search methods are discarded) con- 
tribute to convergence towards the optimum. Their con- 
clusion is that the LMS is a better algorithm; it converges 
faster and produces less steady-state misadjustment. In 
[6] Widrow et al present versions of the LMS algorithm 

that operate on complex data. This concept has recently 
become important because of the use of adaptive techniques 
in the frequency domain, Dentino ines) eer Zentner [17]. 
Lucky, £7], introduces a Minimum Magnitude performance 


criterion to derive an adaptive equalizer. Digital 
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communication systems use equalizers to reduce the inter- 
symbol interference in a communication channel. Lucky's 
solution involves transmission of a special training se- 
quence which is known at the receiver and is used there as 
the reference signal. Another interesting point in his 
solution is the use of quantized variables in the adaptation 
algorithm. 

Finally Frost [26], Owsley [29], Widrow et al [38], 
Griffiths and Jim [41] and many others discuss the use of 
the LMS algorithm for adaptive control of sensor beamforming 
arrays. We will not discuss these applications in this 
dissertation because of their specialized nature. However, 
it is noted that the simplified algorithms presented here are 
general and may be used to advantage in antenna arrays. 

From the references the importance of the LMS algorithm 
is very clear. Surprisingly enough, very little was done to 
improve the basic algorithm, the emphasis being primarily on 
applications of the concept. Gersho [40] discusses adaptation 
in a quantized parameter space. Gersho's discussion is of a 
general nature, i.e. no specific performance criterion was 
assumed, and his main results is that for unimodal performance 
surfaces and deterministic gradient (i.e. no need for sto- 
chastic gradient estimation), the quantized algorithm will 
converge to the neighborhood of the optimal solution. 

Noschner [27] is the only published attempt to derive 
computationally more efficient versions of the basic LMS 


weight adaptation algorithm, and these results have not 
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been used in practice. Griffiths and Jim in a recent paper 
[41] discuss a simplified adaptive system from another point 
of view. Their concern is to simplify the signal processing 
section in order to achieve high frequency operation. They 
propose a 3 level weight quantization, with no multiplica- 
tions in the signal processing portion. The resulting 
weight adaptation scheme is based on the LMS algorithm, and 
it 1S necessary to store past quantizations. Hence it is 
more complicated, but the goal of high frequency operation 
is achieved. 

summary of LMS Algorithm 

Because of its importance, the LMS adaptive algorithm is 
presented here following the basic references [1, 2, 3, 4, 5]. 


The basic filter output is given by: 


a, (kK) x (k-1) (1.2=1) 


where: k is the time index 
NS is the number of filter weights 
a. (k) is the ith weight at time k 
x(k) = s(k) + n(k) is the input signal consisting of 
desired signal s(k) and additive noise n(k). 


We want to minimize the performance function: 


J(k) = Efe¢(k)} = EL{y(k) - s(k)}71 (1.2-2) 


yy 





where: € (x) = yCk) - s(k) is the error 
In order to perform the adaptation algorithm we need the 


gradient of the performance surface: 





1 = O,1,...,N_-1 (L275) 


In practice we don't have J(k) since s(k) is not known nor 
do we have an ensemble of processes to perform the expecta- 
tion operation of (1.2-2). Thus we must use an estimate 


of the performance function: 


J(k) = e2(k) = fy (k) - (xk) }7 (1.2-4) 


where r(k) is a reference signal, not necessarily identical 
Bo s(k). 


The gradient estimate is given by: 


m ea ad€. (k) d€_ (k) 
_ dJ(k) r 2 = 

Nana! ie da, Z das 2e€, (k) da. 
_ d0y(k) _ mer = 
= 26, (K) Sa = 2€, (k)x(k-i) fi 2=5) 
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Using the gradient estimate of (1.2-5), the LMS weights 


adaptation are given by: 


a. Cktl) = a. Ck) = ue ie = a; Cx) -~ 2u£ Ck) xCkx-1) 


die ee a2 irene ois N71 (1.2-6) 


where We 1s the adaptation gain controlling the convergence 
and steady-state properties of the filter. 

Reference [4] assumes a stationary input with uncorre- 
lated samples and derives formulas for the stability region, 
convergence time, and misadjustment as follows. 

Stable convergence of the adaptation algorithm is limited to 


values of We given by: 


o< us, < WINR,, (011 (1.2=-7) 


where R 6m) = E{x(k)x(k-m)} is the autocorrelation function 
of the input. Equation (1.2-7) was derived using the mean 
of the gradient estimate. So, in practice, in order to be 


Stable at all times we need 


We oo NR) i); 
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The approximate Mean Squared Error (MSE) convergence time 


constant is given by 


TusF =1/[4u RR, (0) ] (1.2-8) 


The misadjustment, M, is defined as the ratio of the excess 
Mean Squared Error (MSE), due to adaptive filter steady- 


state jitter around the optimal solution, to the minimum 


Mok : 
_ J steady-state - J min _ 2- 
aoe J min u NLR, (0) (1.2-9) 
where 
Jss = J steady-state = Lim J(k) 
i k > @ 
J min = Jss cof the optimal, = Minimum MSE 


Piicer 


The misadjustment estimate (1.2-9) was derived for an ideal 
reference signal, r(k) = s(k), and does not apply to cases 
of noisy reference. 

The derivations in [1, 2, 3, 4, 5 J are based upon the 
use of eigenvalue eigenvector analysis. To obtain practical 
estimation formulas the Ce based equations are 


approximated by correlation functions. The analysis presented 
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in this dissertation makes the approximations at the start 


of the derivations and uses correlation functions throughout. 
The advantage of this approach is that it provides better 


insight into the nature of the approximations. 
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imoeww lhe TTR*® ADAPTIVE FILTER 
An IIR filter uses previous output values to compute 


the present filter output: 


N71 Ny 


yi) 2 a. * (k-i) + Z b. y (k-i) (1.3-1) 
a : 1 
1=0 i=l 


Because of the feedback in (1.3-1) the impulse response may 
be infinite and is designated IIR. 

Because of inherent savings due to the use of previous 
calculated values (the existence of poles in the transfer 
fminction), the IIR filter is the most efficient filtering 
scheme for many applications. 

Since it uses feedback, the IIR filter can be unstable. 
This presents a design problem for the conventional IIR 
filter, and a basic requirement foran IIR adaptation algorithm 
is to assure that the resulting filter is stable. A second 
disadvantage of the IIR adaptive filter is the multimodal 
nature of its performance surface as discussed in section 3.1 

White [8] was the first to suggest the use of IIR struc- 
tures for an adaptive filter. He indicates a possible use of 
several performance criteria and derives the gradient ex- 
pression for the Minimum Mean Squared Error (MMSE) performance 
criterion. In [9], Stearns et al presents an all adaptive IIR 
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* FIR (Finite Impulse Response) and IIR (Infinite Impulse 
Response) are generally used by the signal processing community 
to denote non-recursive and recursive filters respectively and 
SeensO used in this-work. It is noted though, that some 
reeursive filters can have a finite impulse response. 
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filter. Stearns' algorithm is rather complex, i.e. the number 
of operations (multiplications and additions) is proportional 
to a,NN ta5N,*, compared to the relative simplicity of 

the LMS where the number of operations is proportional to Nu: 
Stearns' algorithm is discussed later and its gradient esti- 
mation method is presented with details. 

In [10] Feintuch presents a much simpler adaptive IIR 
filter which consists of two LMS adaptive sections, one controls 
the feedforward weights adaptation and the second controls the 
feedback weights adaptation. Feintuch's algorithm gradient 
estimation method is presented later on in this section. 
Feintuch's algorithm works in some cases but, as pointed out 
by several investigators [11, 12], the derivation has errors 
and the filter, at least in the examples presented in [11], 
does not converge to the optimal solution. 

In [13] Parikh and Ahmed used the same examples presented 
in [11] to demonstrate the convergence properties of Stearns' 
algorithm. Reference [13] shows that Stearns' algorithm does 
converge to a minimum point, but with a multimodal performance 
surface the steady-state might be around a local minimum or the 
global minimum depending upon the starting point of the adapta- 
tion process. McMurray, [14], investigates the dependence of 
Feintuch's algorithm stability on the values of its adaptation 
gains. The region of stable operation turns out to be a tri- 
angle in the adaptation gains space. In [15] McMurray inves- 


tigates the convergence time for Feintuch's algorithm IIR 


filtering of narrow band signals and compares operation in the 
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time and frequency domains. In both cases the convergence 
time is inversely proportional to the square root of the 
multiplication of four factors: feedforward adaptation gain, 
feedback adaptation gain, number of feedforward weights, and 
the number of feedback weights. An additional conclusion was 
that the convergence time is shorter for the time domain 
operation. Parker and Ka, [18], extend the adaptive IIR 
filter for image processing. In [35] Treichler, Larimore 
and Johnson modify Feintuch's algorithm by passing the error 
term through a FIR filter. This modification allows for con- 
vergence to a minimum (not necessarily global), and its use is 
limited by the information needed for the design of the error 
term filter. The existing IIR adaptive algorithms are based 
upon Stearns! and Feintuch's algorithms which are summarized 
briefly in the following. 

In order to have a practical adaptation method we use a 


performance function estimate: 


S (k) = ean) = {y(k) - a) b= (1.3-2) 


where r(k) is the reference signal and y(k) is given by 
(1.3-1) with weights a, (k) and b, Ck) being a function of time. 


The gradient estimate is given now: 











n a de. (k) 
va, = 2k) = 29¢ —Z = 2c WN Gi ey) 
14 das r a. a 
ae ay(k) . ; 
the derivitive aS is not as simple as in (1.2-5) because 
i 


of the feedback terms such as baylke-9) present in y(k). 
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The final form of the gradient estimates of Stearns' algorithm 


are given by: 


Bat = Ze. (k) Os (k) i= 0,1,...,N-1 (1.3-4) 
where: 
Ny 
a, (k) = X{k-1) ee Pg Wis) ey Masa), (1.3-5) 
and: 
eat = 2e_(k) 8, (kK) i= 1,2,...,N, (1.3-6) 
where: 
Np 
B, (k) = y(k-1) gh b (kc) Bs C=) ; (i238 =7) 


Equations (1.3-4, 5, 6, 7) are the gradient estimates of 
Stearns' algorithm. Feintuch's, [10], algorithm uses only 
the first terms in the expressions for Os (1.3-5), and BS 


(L.3-7) and the resulting gradient estimates are: 


<I> 
~~ 
ll 


Ze (k) Xe i) (i. 3=39) 


Vp, (k) = 2e,(k) y (kod) (1.3-9) 


With both algorithms the weights adaptation is given by: 


INS; 
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where We and HW, are the feedforward and feedback adaptation 
gains. These adaptive IIR filtering schemes are not satis- 
factory solutions to the IIR filtering problem. Stearn's 
algorithm is not satisfactory because of the following 
reasons: 
- The instability problem mentioned by Elliott, Jacklin 
and Stearns, [25]; this problem is discussed later. 
- The algorithm does not assure convergence to the global 
minimum. 
- It is a complicated algorithm. 
The Feintuch algorithm is not satisfactory mainly because, in 
some cases it fails to converge to a minimum point, and in 
all cases does not assure convergence to the global minimum 


of the performance surface. 
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it ONTRODUCTION TO ADAPTIVE FIR FILTERS USING SIMPLIFIED 
GRADIENT ESTIMATIONS 


The LMS algorithm is being used in many adaptive fil- 
Zane applications Lles6, 16, 17, 22, 24, 26, 29, 32, 34, 37, 
38, 39, 411, with satisfactory results. The possibility of 
using simplified algorithms, with hardware and time savings, 
has not received much attention. Gersho avons and Moschner 
[27], and recently Griffiths and Jim [41] (which discusses a 
somewhat different problem of simplifing the signal processing 
portion with more complicated adaptation algorithm) appear 
to be the only publications in this area. All applications, 
except [41], seem to select the classical LMS algorithm and 
not a simplified version. A possible reason for this fact 
might be the lack of confidence in the performance of a 
simplified algorithm, compared with the many satisfactory 
results obtained with the use of the LMS algorithm. This 
dissertation will demonstrate analytically, and by extensive 
Simulation, the advantages and savings associated with the 
use of the simplified algorithms. One natural simplified 
algorithm investigated here is the use of a positive or nega- 
tive Fixed Step Correction (FSC) in the adaptation, instead 
of the LMS correction which is proportional to the value of 


the gradient. This gradient estimation is given by: 


as ~ 


eae) = SgntV sug (ik) J=Sgnte (k) }Sgn{x(k-i) } (1.4-1) 
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where: 


| Petes CO 
Sombie — ed 
at ee sO 
The second algorithm investigated here is to use a modified 
FSC with the step size proportional to the magnitude of the 
error. This algorithm is called here the Simplified LMS 


(SLMS), Moschner [27] called this the clipped LMS. The SLMS 


has the following gradient estimate: 


las 


Vorms (ik) = € |. (ik) Sgn{x(k-=-i) } (1. 4-2) 


Chapter II discusses these algorithms and presents an analysis 
and simulation of adaptive FIR filter operation using these 
algorithms. 

The optimal Wiener filter depends upon the statistics of 
the input signal and the desired signal, the steady-state be- 
havior of an adaptive filter depends upon the corresponding 
statistics. Since the desired signal is not available for the 
adaptive filter, and it uses a reference signal which is only 
related to the desired signal, it is obvious that the pro- 
perties of this filter differ depending upon the application 
and manner in which the reference signal is provided. In 
Chapter II an adaptive FIR filter is used as an adaptive line 
enhancer (ALE) [3, 34, 37, 39] which is a typical signal 


processing application and utilizes a noisy reference. 
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Appendix A describes the simulation details. 

The discussion in Chapter II includes for each algorithm 
ine following topics: 

- convergence and stability, section 2.2. 

- convergence time (TC), section 2.3. 

- steady-state misadjustment (M), section 2.4. 

- implementation complexity, section 2.1. 

- dynamic range, seciton 2.6. 

Sections 2.3 and 2.4 include derivations of estimation 
formulas to the convergence time, TC, and misadjustment, M, 
of the FSC and SLMS algorithms. 

The simulation experiment, described in Appendix A, shows 
good agreement to these misadjustment and convergence time 
ee ulas. 

Fig. 1.4-1 presents a typical operation of the adaptive 
mm filter with LMS, FSC, and the SLMS algorithms. This 
figure shows a typical weight, a1» and the Mean Squared 
Error (MSE), as a function of time for the three algorithms 
as noted. On each plot we have drawn the optimal value of 
the weight or the MSE, an ensemble average of 100 runs as well 
as the convergence of an individual filter (single run). In 
Fig. 1.4-1 all of the algorithms perform, on the average, 
about the same. 

For more accurate comparison, a graphic presentation of 
adaptive filter properties is introduced in Section 2.1. This 
graphic presentation, the Adaptive Filter Operating Charac- 
teristic (AFOC) is used to compare equal degree and equal 


Complexity filters with different algorithms. 
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If one looks ahead to Fig. 2.1-l1 it is apparent that the 
simplified gradient algorithms (FSC and SLMS), when compared 
to the LMS algorithm with equal complexity (cost) and equal 
convergence time, are more effective and provide more pro- 
cessing gain (processing gain is defined later as a measure 


Sr filter effectiveness). 
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1.5 INTRODUCTION TO ADAPTIVE IIR FILTERS USING RANDOM 
SEARCH TECHNIQUES 


Adaptive IIR filters based on gradient methods have one 
major disadvantage which is the multimodal structure of the 
performance surface as discussed in section 3.1. Thus there 
is no inherent way to assure a steepest descent gradient con- 
vergence to the global minimum. The convergence problem 
and additional disadvantages of Stearns! and Feintuch's 
algorithms, as discussed in section 1.3, suggests that gra- 
dient methods may not be the best adaptation scheme for the 
IIR filter. Thus a different adaptation technique, namely, 
random search, 1s considered here. The basic concept of 
random search is discussed in section 3.2. 

A random search IIR filter is presented and discussed 
in section 3.3. It is concluded there that this scheme is 
not satisfactory. The fact that the IIR filter's performance 
surface is quadratic in the feedforward weights (Elliott et 
al L25]) is the key for the hybrid Random and Gradient 
Search (RGS) algorithm developed in section 3.4. This new 
algorithm provides for satisfactory operation of an IIR 
adaptive filter. Convergence analysis of the RGS algorithm 
and convergence time estimation for a typical signal pro- 
cessing situations 1s given in section 3.5. 

For cases where information is available on the structure 
of the optimal filter, a constrained, or apriori structured 
filter algorithm can be implemented. This concept is 


discussed in section 3.6 and shows good results. Fig. 1.5-1 
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presents the error convergence of four filters - the LMS FIR 
filter with 20 weights, a RGS IIR filter, an apriori struc- 
ture adaptive pole (ASPOL, section 3.6) IIR filter, and 
Feintuch algorithm IIR filter. The IIR filters have two 
feedback and three feedforward weights. For this example it 
is seen that: 
1. The LMS algorithm converges fastest. 
2. The RGS converges slower but reaches a lower steady- 
Sead Eeuwernor:. 
3. The ASPOL converges to the lowest steady-state error, 
faster than the RGS. | 
4. Feintuch algorithm converges to the highest steady- 
state error. 


These examples are typical. 


33 





ERROR CONVERGENCE FOR SEVERAL ALGORITHMS 
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Di eoOLone Levees RRO ERETERS USING SIMPLIFIED 
GRADIENT ESTIMATIONS 


2.1 TWO SIMPLIFIED GRADIENT ALGORITHMS 
Two simplified gradient algorithms are considered: 
(a) The Fixed Step Correction (FSC) adaptation scheme 


1s given by: 


a. (k+1) = a. (k) — u_Sgn{V (iigse 1c) (2.1-1) 


This formulation is essentially binary and was motivated by 
the general success of bang-bang type controllers. The 
adaptation gain Wo is the size of the fixed correction step. 
We define the FSC gradient estimate as: 


“nw 


V Cis oes Sen eco k)}=Sgn te(k)} Sgn {x(k-1)} 


BoC 


(2.1-2) 
Mmshould be noted that the sign of the gradient, Sgn {V (i,k)}= 
Sgn {x(k-i)} Sgn, fe(k)}, is identical for both error magnitude 


and mean square error estimates, that is 


aie | 
Sgn es oe [= oom je , so that (2.1-2) can be 
at at 


derived from either error magnitude or mean squared error. 


large correction steps result in fast convergence to the steady- 
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state near the optimal filter weights with a large steady- 
state jitter around the optimal filter. These contradicting 
effects call for engineering compromise in choosing the size 


of the correction step im . 
(b) The second approach is to use a variable size 


step. A natural possibility is to consider 
ieee 2” |e (k) | (eles) 
Mae combination of (1.2-2), (2.1-1), and (2.1-3) gives: 


a. (k+l) = a. (k)-u'|e (k) | Sgn {e(k) }Sgn{x(k-i) } (2.1-4) 


We can use the regular adaptation gain symbol He instead of 


mM’ and write 


a, (k+1) = a. (k) -H_€ (k) Sgn {x(k-1) } (2.1-5) 


1-5) is the simplified LMS (SLMS) algorithm with the 


gradient estimate given by: 


v : 2 -3 pec 
Vorms (t+) e(k) Sgn {x(k-i) } | ( ) 
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Typical operation of the LMS, FSC and SLMS algorithms are 
presented in Fig. 1.4-1l. 

A useful graphic presentation of adaptive filter proper- 
ties is given by a plot of processing gain (PG) as a function 
of convergence time (TC). The processing gain measures the 


Filter effectiveness and is defined as: 


PG = 10 loglR7(o)/J..] (2.1-7) 


where Rin'?? is the input noise power and lees defined in 
(1.2-9), is the output error power. 

The convergence time, TC, is the time required to reduce 
90% of the initial excess MSE. The value of the performance 


function at the time TC is: 


J(TC) = ue pes 10s eel. CO)) eo = J 551 (2.1-8) 
This plot, named the Adaptive Filter Operating Character- 
istic (AFOC), can be used for design when the number of filter 
weights, No? 1s a parameter. It also provides a method of 
comparison for different adaptation schemes. Curves for the 
LMS, FSC, and SLMS algorithms are presented in Fig. 2.1-1A 
for the ALE experiment of Appendix A. 
We define the following complexity index (CF) for com- 


paring adaptation schemes. 


a= Oh | +o (2 cea) 


2 Napp 3 Neon 
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where NwuL>? Napp? Noon 2remuncenumber Of multiplication, 
addition and control operations used in one iteration. Os > 
G5» 2 are weighting coefficients representing the cost of 


each operation. A reasonable approximation which neglects 


control operations is 


cr = ao N + N (2.1-10) 


Using the equations for LMS, FSC and SLMS techniques we have 
the following complexity indices as a function of the number 


of the filter weights, No: 


CFius = (2N + l)a + 2N,+ 1 (2.1-11) 

CFigo = Nia + 2N_+ 1 (2.1-12) 

CFomMs v= (N+ eee ne 2N * iL (251=13) 
As a reasonable numerical example, using a = 5, we have 
approximately equal complexity with Nous =e Nege lal 


Nora = 10. The AFOC comparison for this complexity is 
presented in Fig. 2.1-1B and indicates that for a given 
convergence time the simplified gradient methods provide 


higher processing gain. 
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42.2 CONVERGENCE AND STABILITY 

In this section we discuss conditions for the convergence 
and the stability of the simplified gradient estimates. A 
stable adaptive filter is one that converges to a near 
optimal steady-state. We now define the convergence ratio, 


em. (k): 
4 x 
a. (k+l) ~ a, 
SS (2.2-1) 
i 
am) tas 
1 1 
x 
where as is the optimal value for the weight a.- 


Following Widrow et al [l, 3, 4] we define the weight 


noise, V,(k), as: 


x 
V, (kK) = a, (kK) - a; (22— 2) 


Combining (2.2-1) and (2.2-2) gives: 





V; (k+l) 
C, (k) = ~V, (kK) (2.2-3) 
From (2.2-2) and (1.2-6) we get: 
V, (k+l) = V, (kK) _ ace (2.2-4) 
Combining (2.2-3) and (2.2-4) we get: 
Va.(k) 
-i1- sat = 
Cc. (k) 1 le Tw (2.2-5) 


The steady state average convergence ratio is defined as: 
C; = Fic, (k) } = 1-uE yy’ ; (2.2-6) 
where k is large enough for operation of the filter to be in 


Steady-state. From this point we proceed with the specific 


Case of the SLMS, with its gradient estimate given by (2.1-6). 
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The error as function of the weight noise is given from (2.2-2) 


and (1.2~4) as: 


* a 
é (kK) = € (Kk) + B Vayin8 (k=) (222-7) 


* 
where ey(k) is the optimal error at time k and is given by: 


# _ a x 
E(k) = : as M(t koa) (kK) (2,.2-8) 


Inserting (2.2-7) to (2.1-6) results in the equation: 


a * 


Vorms (t+) = €L(k) Sgn {X(k-i)} + 


V,(k) X(k-J) Sgn {x(k-i) } (2.2-9) 


Inserting (2.2-9) into (2.2-6) we get: 


* 
¥ —€(k) Sgn{x(k-i)} 
e=)] =< yu liye 


a a V, (k) 
eos Vek) | 
+ 2 E Gags) - X (k-j3) Sgn {x(k-i) } }} (2.2-10) 
j=0 i 


* 
c. (k) is independent of x(k-1i) and of V, (k) so that: 
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E Uk) Sgn{x(k-i) } 
7 Sgn{x(k-i) } Z 
E aaa Ble. (k) }Et V,(k) }=0 (2.2-11) 
because ELe™ (k)] = 
To continue with the simplification of (2.2-10) we make the 


following assumptions: 


(a) V.(k) and x(k-j) are uncorrelated 
: } CLD, 

(b) BtV. (k) /V, (k) = 1 
Assumption (a) is similar to the uncorrelated input assump- 
tion used by Widrow in [1] and seems to be justified by his 
results. Assumption (b) is made for mathematical convenience 
and can be justified by the dependence of the weight noises 
on the common error terms and the uniform statistics of the 


input signal over the filter memory. 


Using (2.2-11) and (2.2-12) in (2.2-10) we get: 


N_-1 
a 
EZ Ef{x(k-j) Sgn {x(k-i) ]} (2.2-13) 
j-o 


Since x(k-j) Sgn {x(k-i)} < |x(k-j)| we can write: 


2 
© <1l-w. £ fs |x(k-3)| (2.2-14) 


For stationary input signals E{x(k-j)} = Ef{x(k)} with all 


values of j and we get: 
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C, < 1 - un, El x(k) | (262-15) 


For stable operation, as well as for convergence to the 


optimal weight values, we require 


jc,| <1 (2.2-16) 


Manipulating (2.2-15) and (2.2=-16) to obtain the stability 


condition for ue yields for the SLMS algorithm: 


2 


To express (2.2-17) as function of the input power, Rye (0) 


we can define the input signal form factor, Fo as: 
P= E|X(k)|// E(x? (k)} (2.2-18) 


Now inserting (2.2-18) in (2.2-17) results in: 


me< SC ane he (2.2-19) 
SLMS NFY R,, (0) 


For the LMS algorithm we can use (2.2-6) and the LMS gradient 
estimate. Following the above derivation and using the 


assumptions of (2.2-12) we get 
(2.2-20) 
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(2.2-20) is equivalent to equation (32) in [4] which was 

derived in a diferent manner but with similar assumptions. 
To derive the stability region of the FSC algorithm we 

use (2.2-17) and the relationship between the FSC and the 

SLMS algorithms, we define an equivalent adaptation 

gain, Meg? by the formula 


laee Mage! €, (K) | (2.2-21) 


It is interesting to note that we are now using the deriva- 
tion process of Section 2.1 for the SLMS algorithm in a 
reverse direction. The case of greatest interest is that of 
a low signal to noise ratio. For this case we use the 


following approximations: 


€ (Kk) = y(k) -s (k+l) -n(k+1) ~-n(k+1) *-r (k) 
and 


Ble (k) |*E[xr(k) | = E| x(k) | (2.2-22) 


/ 


Inserting leg from (2.2-21) into the SLMS relation, given by 


(2.2-17), with the use of (2.2-22) results in the following. 
oO < Unsc < 2/N. (2.2-23) 


The foregoing relationships (2.2-17), (2.2-20) and (2.2-23), 


are based upon average behavior of the algorithms. In 
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practice, to avoid numerical overflow, we must use adaptation 
gain values much smaller than the upper limit indicated in the 
above relations. An additional consideration that also results 
in a smaller adaptation gain is the misadjustment. For all 

the algorithms, the use of the upper bound value for the 
adaptation gain results in a misadjustment of the order of the 
optimal filter gain (PF), which means that practically we are 
restricted to much lower values of the adaptation gain. The 


results of this section are included in Table 2.6-l. 
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2.3 CONVERGENCE TIME ESTIMATION 

In order to estimate the convergence time of an adaptive 
filter one may visualize the process as changing the weights 
with some average step, A, taken in most of the iterations 
towards the optimal value of the weight. Assuming an initial 
value of zero for all the weights, the longest convergence 
time will be associated with weight having the largest abso- 
lute value, a - From the above it is reasonable to assume 


max 
the following relationship: 


O 
TC = e271 =e : No (2.3-1) 


where: N. is tne number of filter weights, and Gi, G5 are 
unknown coefficients. anax/4 is the exact number of steps 
needed for convergence if the correction is always in the 
right direction. In practice the gradient estimation causes 
errors in the direction, and the number of iterations re- 
quired to converge to the optimal value of the weights is 
modified by a factor that depends in some non-linear way on 
the number of weights N_: This modification is represented 


o 
in (2.3-1) by the factor a; N 2 Also Oy depends on the 


exact definition of TC (i.e. 10% or et 


of the initial error 
Squared). Filter operation involves a linear combination of 
input values. Since the reference amplitude is independent 


of Noe when we combine more input samples the relative weight 


associated with each sample should be smaller, mathematically: 
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anax  N_ eee 
In general, do, depends upon the input signal to noise ratio 
as discussed in the literature [3, 33]: This dependency is 
not taken into account in the derivations which follow in 
order to Sisgres ie comparison of the new algorithms with 
existing algorithms. The results of reference [37] can be 
used to modify the results presented here to include the de- 
pendence upon input signal to noise. : 

When looking at specific applications, such as Adaptive 
Line Enhancement (ALE), one can determine the value of om in 


(2.3-2) exactly. Inserting (2.3-2) to (2.3-1) and absorbing 


om into a), we write: 


or 
oT N,? 


TC = = ANTS (263-3) 
a 
A in (2.3-3) depends on the adaptive scheme. It is the 
fixed step size in the FSC algorithm and an average step 
size for the LMS and the SLMS algorithms. Thus for these 


three cases we define: 


Fsc "“rFsc cea 


w~ 


ums = EllUrms Yeas |} = 2Upas 


> 
| 


E{|e(k)x(k-i)|} = (2.3-5) 


sums = ={l gps "stud Horms®{ lek) Sgn x(k-i)}|}  (2.3-6) 
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Using (2.3-3) and (2.3-4) and the empirical coefficients a= 


1.65 a, = 1/2 as evaluated using the simulations described in 
Appendix A, we get the following FSC convergence time to 10% 


of the initial squared error: 


To = 2S) CG GEDS 


Haga MN, 


where TC is the time required to reduce the error to 10% as 
Mefined by (2.1-8). Fig. 2.3-l1 presents a verification of 
(2.3-7) using simulation results with several values of Ursa? 
No and the input power R60) > 

The significance of these results is that they confirm 
that the convergence time is inversely proportional to the 


adaptation gain and the square root of the number of weights. 


Assuming in (2.3-5) that - 


E{|e(k)|-:|x(k-i)|} = Ele(k)|- E|x(k-i)| we get: 


A Ele(k)| E|x(k-i) | (2.3-8) 


ums ~ *4tms 


At the start of the adaptation process the initial weights 
have a value of zero, so that y(0) = 0 and ¢«(0) = r(0). For 
the correlated reference case the reference power 1s essen- 


tially the same as the input power and we have: 
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A E|x(k)| ° E|x(k) | (2.3-10) 


ims ~ 74tMs 
Using expression (2.3-10) for the LMS average step size in 
(2.3-3) with a, = 1/2 we have 


0 
ZC a rar (2.3-11) 


LMS 9) 
Urs (B1x (6) [2° A 


In [4] the classical LMS convergence time estimate is given 


by: 


ime ako 
_ (oe s= 1) 
4 Urs Rye (9) 


Based on the simulation described in appendix A we select 


MS = 7.555. 


Fig. 2.3-2 presents a comparison of simulation results 
with the classical convergence time formula, (2.3-12), and 
the new convergence time formula, (2.3-11). This figure 
indicates clearly that the convergence time depends upon the 
number of weights, No: as developed in (2.3-11), and that 
this formulation is more accurate than that of (3.2-12) which 
was developed in reference [4]. In a similar way (2.3-6) and 
(2.3-3) gives 


0 
TC ee (2.3-13) 


Usrms NEI x(k) | 
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Fig. 2.3-3 presents a comparison of (2.3-13) with simula- 
tion.results, with Aorus = 1.4, based upon the results of 
simulations described in Appendix A. The comparison confirms 
(2.3-13). The key formulas of this section are included in 


Table 2 e G1 8 


a2 
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2.4 STEADY STATE ERROR AND MISADJUSTMENT 
In order to evaluate the steady-state error we start with 
general relationships. First following [1, 4, 3] we define the 


weight noise v , Ck) as: 





* 
| v,(k) = a,(k) - a, (2.4-1) 


~— 


where a. is the optimal ith weight. 


Ne—] N_-L 
a a a 
yv(k) = = a.(k) x(k-i) = IE a. x(k-i) 
i=0 * i=0 7+ 
Nea 
a 
ee 2 v.(k) x(k-i) (2.4-2) 
i=0 * 
Define 
N_-l 
ch * 
eo (k) = y(k)- s(k) = £ a, (k) x(k-i) - s(k) 
L=9 
i Ne 
a 
+ v.(k) x(k-i) 2.4- 
i=0 7 | i 


We can now define the Optimal instantaneous error: 
& = & 
meik) = 2} a. x(k-i) - s(k) (2.4-4) 


Using this value the minimum mean squared error is: 
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* 
J. =E fe ce with k in the steady state. (2.4-5) 
min Ss 


From (2.4-3) and (2.4-4) it follows that: 


2) J(k)| k in the steady state 
. N71 
= E{{e_(k) + 2 v. (k) x (k-i) }*] 
1=0 = 
A Nit i 
= E{e (k)} + E{2 <£ v.(k) e€_(k) x(k-i)} 
S mae a S 
1=0 
N_-1 N -l 
a a 
+E in x | z Ve (k) Ve (k) x(k-i,)x(k-i,) } 
1-0 1=0 uf 2 


(2.4-6) 


The foregoing assumes: 


(a) The expectation of v, (k) x(k-1) 1s factorable. 


gee 
(b) Etv, (k)v. (k) J = 4 S35 
i 1 as) 
WheEBemOs a. = 91 
i 0 iF 3 


Assumptions (a) and (b) appear to be well justified in the 
case of a correlated reference signal,:. as. confirmed by the 
agreement obtained between the derived formulas and the 


Simulation results. 


oye, 








The second term of (2.4-6) can be factored: 


* e 
tv, (k) E , (k) x (k-i) } 


E{v, (k)}: E{e_(k) +x(k-4) } (2.4-7) 


| 
* 
However, Efe. (k) x (k-i) } 


O (because of the orthogonality of 


{ 


the optimal solution) so that the second term is zero and 


(2.4-6), using assumptions (a) and (b), becomes: 





N -l 
a 
5 ie eer 
ss min ae 
1=0 
= me, 2.4-8 
Jmin bee Re (0) = 
We now define the excess MSE as follows: 
2 
J, = ers - Jin = NV Sra) (2.4-9) 
and the Misadjustment as: 
2 
a Nov Rye (9) 
M= Te = aan, ane (2 20— 109 
min min 


2 
, Ni,V , and Ry (0) - Ry (9) 


depends upon the statistics of the input. Seats depends upon 


the input statistics as well as N.. However, v- depends upon 


the foregoing depends upon oan 


the nature of the specific algorithm and will now be considered 


for the SLMS and FSC algorithms. 
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From (2.4-1) and (1.2-6) we have: 


v; (k+l) = v, (k) = fea (2.4-11) 





Squaring both sides we get: 


2 aw. 2 7 eRe A : 
v5 (k+1) = V5 (k) + uy 7. (k) 2u iv, (k) Ya," (2.4-12) 


For the SLMS algorithm: 


v,(k) ¥, (k) =v, (k) €,(k) Sgn {x(k-i)} = 


aL 
N-1 
= v, (k) Le, (k) + £ v4 (k)x(k-Jj) } Sgn {x(k-i)} (2.4-13) 
d= 
where 
€.(k) = y(k) -~ x(k) (2.4-14) 
and 
N -1l 
* a * 
e (k) = 3 a. x(k-i) - r(k) (2.4-15) 
r 4=0 ie 


* 
€ (k) and € (Kk) depend on the reference signal r(k), and in 
Many cases, including the correlated reference case, those 
* 
errors and the previously defined E 5 (k) and € , (k) have 


completely different statistics. 
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* 
v, (k) and x(k-i) are independent of E(k) SO we have: 


* ' = 
E tv, (k) €) (x) Sgn {x(k-i)}} = 


E {e,(k)} + E{v,(k) Sgn {x(k-i)}} = 0 (2u4216) 

we get: 
N71 
Biv (ej Vick) ) = 2 E{v. (k) v. (k)x(k-j) Sgn{x(k-i) }} 
al a. 6 i 5 
(2.4-17) 

Using assumptions (a) and (b) in (2.4-17) we have: 

ELv, (k)V,'(k) ] = = E| x(k) | (2.4-18) 


1 


taking the expectation of (2.4-12) in the steady state and 


using (2.4-18) we have: 


El vs (k+1) J=ELv4 (k)] + u2E{e2 (k) Sgn7Cx(k-d) J} 


2 


- 2u_v E|x(k) | 


(2.4-19) 
In the steady state Elv{ (k+l) J = ELv¢ (k) J and 
eo) 
from (2.4-19) we can express v’ as: 
—z u_E [e*(k)] 
2 a © 


ye . (2.4-20) 
2 E|x(k) | 
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Now we can insert (2.4-20) to (2.4-10) and get: 


2 
u_N_ELe. (k)] R._ (0) 
4 Q J eas (2.4-21) 
2 = BK (K) | 
min 


BLES (k) J Gepends upon the type of reference used. For the 


correlated reference we have: 
€ (Kk) = y(k) - r(k) = y(k) - Us(k+l) + n(k+1)] (2.4-22) 
By squaring and taking the expectation of (2.4-22) we get: 


Ele@(k)] = E{ly(k) - s(k+1)]°} + Bln“ (k+1)] 


— 


= 2E fy(k) n(k+l) - s(k+l) n(k+1)}  (2.4-23) 


In the third term of (2.4-23), n{k+l) is independent of s(k+l), 
and the present output y(k) is independent of the future noise 
n(k+l), so this term's expectation is zero. Because of the 
correlation of s(k+l) and s(k), which is a basic requirement 

| for the use of the correlated -reference, the first term of 


(2.4-23) will be: 


E{[y(k)-s(k+1)]°} = E{Ly(k)~s (k)]7}= Ele“ (k) ]= a 


(2.4-24) 
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For stationary noise we have 
2 2 
Ron ‘9? = E{n’(k)] = Eln’ (k+1) ] (2.4=-25) 


Using (2.4-24) and (2.4-25) and the above reasoning about the 


3rd term, (2.4-23) becomes: 
2 — es 
E Le, (k) J SS hems Wee ne.) (2.4-26) 
For reasonable processing gain J, <<R,, (0) and 
gE fe“(k)] = R__ (0) (een) 
e nn ° 
The optimal processing factor (PF) of a filter is defines as: 
oS a (2.4-28) 
PF express the optimal noise reduction possible by an optimal 
filter of order N.- PF depends upon No and the signal sta- 
tistics, and does not depend upon the adaptation algorithm 


and the adaptation gain. 


Inserting (2.4-27) into (2.4-21) we get: 


BeN. RR. (0) Ry, (9) 


aaonn 
m=. (2.4-29) 
2 E| x(k) | 2 Bares 


Using definition (2.4-28) in (2.4-29) we get: 
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M = sETeT > PF (2.4-30) 


Fig. 2.4-l1 presents a verification of (2,4-30), using simula- 


tion results with several values of Hor ye? No» and the input 


———— 


_ power R60)» 

Equation (2.4-30) was derived for the SLMS algorithm. To 
derive an equivalent expression for the FSC algorithm, we use 
(2.4-30) and the relationship between the FSC and the SLMS 
algorithms given by (2.2-21). The case of greatest interest 
is that of a low signal to noise ratio, for this case we can 
use the approximation given by (2.2-22). Inserting Meg from 
(2.2-21) into the SLMS relation, given by (2.4-30), with the 
use of (2.2-22), results in the following: 


Hag ch Rex (9) 


M = 5 
E(k) | 4 


PF (2.4-31) 


Equation (2.4-31) provides an estimate of the misadjustment 
of the FSC algorithm and Fig. 2.4-2 illustrates it's agree- 
ment with the simulations. It should be noted that because 
of the approximation of (2,.4-31), the accuracy of (2.2-22), and 
the accuracy of (2.4-31) should improve for lower signal to 


moise ratios. 
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A. SLMS MISADJUSTMENT YSU, ,Noi@ ,RXX(O)=1 


oe” 
e® 
oe 
o* 
a® 
o- 
a®@ 
ef 
o® 
eo? 
e® 
a® 
ae 
Sus 
of 
eo? 
oe 
a® 
ae? 
a® 
e 
ef 
= 
ef? 
a? 
e 
«® 
oe 
. 
ao” 


@.008 0.091 3.00 ‘ : 
2 8.803 0 aie Q@.206 0.007 O.908 6.909 9.919 


B. SLMS FUS NM .U_*©.885 ,RXX(8)92 








5 10 1S 20 25 20 35 
N 


C. SLMS M US RXXCO) ,N«te ,_».e@5 


@ ie co 38 48 Se 69 78 88 
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LEGEND 3 
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Fig. 2.4-1 


SLMS Misadjustment, Simulation Results and Theory 
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A. FSC MISADJUSTMENT VSL, oMH2tO ,RXX(CQI04 






Q 
9.000 0.001 0.002 0.003 09.004 0.895 0.9006 9.007 8.008 0.009 0.019 


a 
B. FSC MUS NH i i*.00S ,RXX(A)94 


5 ie iS 28 es 38 35 


N 
C. FSC MUS RXX(@) ,Notd ,H 58.005 

6¢ 
"40 
OP ec ecco af hte Pte 

Q Tis 
@ 100 eesti CSCC HCH 
RXX(O) 
LEGEND : 
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Fig. 2,.4=-2 


FSC Misadjustment, Simulation Results And Theory 
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2.5 DESIGN CONSIDERATIONS 
The design problem of a FIR adaptive filter involves the 

following major points: 

(a) Selection of an algorithm: LMS, FSC, SLMS. 

(b) Determination of the order of the filter, No 

(c) Determination of the adaptation gain, Wo: 
The discussion which follows does not consider the following 
points: 

(a) A possible IIR filter solution. 

(b) Implementation details. 

(c) Minimization of a given design criteria, such 

aS: cost, volume, weight ... etc. 

The adaotive filter is uSually part of a larger system which 
sets its design requirements. The adaptive filter specifica- 
tions that we consider here are: a desired processing gain 
and an upper limit to the convergence time. The additional 
information required for the design iS some specification of 
the expected input signal to the adaptive filter. Realizing 
that a complete analysis of adaptive filter behavior is not 
possible for complicated signals, we consider a design proce- 
dure based upon simulation and a graphic presentation of the 
adaptive filter properties, the adaptive filter operating 
Characteristic (AFOC) as defined in Section 2.1. As an 
-example, we considered enhancement of a single sine wave of 
unknown frequency with a signal to white noise ratio of 0 dB. 
The desired processing gain is 8 dB and the allowed convergence 


time is 100 iterations. 
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Table 2.5-l1 outlines the suggested design procedure and 
presents the application of this procedure to the foregoing 
example. Since we do not specify implementation details, 
step (6) of the procedure cannot be carried out for the 
example. Hence the example is done for the SLMS algorithm 


only. 
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Scep 


1 

Test 
Signal 
Selection 





2 
Simula- 
tion 


Table 2.5-l 


FIR Adaptive Filter Design Procedure 


Description 


Define a test signal (or 
several test signals) for 
which the filter performance 
will be evaluated. The dyna- 
Mic range of the input signal 
should be considered at this 
point, and might influence 
the selection of the test 
Signals. The test signal 
might be average, worst case, 
or several typical signals. 


Use simulation to generate an 
Adaptive Filter Operating 
Characteristic (AFOC) with 

N aS a parameter for the 
LMS, FSC, SLMS algorithms 

for each of the test signals 
selected in step l. 


For each of the algorithms 
draw, on the AFOC plots, 
lines for the desired pro- 
cessing gain and conver- 
gence time. For each al- 
gorithm select the small- 
est number of weights that 
meets the requirements. 

At this point the designer 
might consider trade-off 
in Noe PG, TC. 





Since each curve on the 
AFOC is constructed for 
several values of u_, one 
can use this data afid the 
values of N_, TC and PG 
selected in “step 3 to 
determine the appropriate 
value for the adaptation 
gain. 
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Example 
Sine wave plus white 


noise with signal to 
noise ratio of O dB. 


In Fig. 2.5-it we 


select Ni=14 


Fig. 2.5-2 we select 
H=.004. 








Table 2.5-1 continuted 


Description Example 





Using the information from all] Fig. 2.5-3 we have 
the test signals we need to:, PG=7.9 dB, TC=114. 
(1) Select a single N_ and u This performance is 
for each algorithm. = marginal and a 

(2) For the three algorithms higher order filter 
evaluate the performance with | should be considered 
this N#, u*~ for each test 

signal: = 

(3) Determine for the LMS and 

SLMS algorithms whether ad- 

justment for dynamic range is 


required. 


At this voint in order to 
complete the specification of 
the filter the designer can 
compare the resulting com- 
plexity of the three candi- 
dates and select the best one. 
The decision depends upon 
implementation details. 
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SLAS AFOC ,RAX(O)"2 ,SAR7U08 


wo te OU 


required convergence time 


No=5 
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Fig. 2.5-1 


AFOC For The SLMS Algorithm, With Details Of The Design Ex- 
ample. Input Signal Is A Sine Wave Plus White Noise With 
SNR= O dB. 
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Convergence Time As Function Of Adaptation Gain For SLMS With 
Ni=i4. For TC=100 We Select Wi =-004 
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SLMS ! NA=14 La*.004 
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Typical Operation Of The Filter With The Selected 
Paramecers. 
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2.6 CONCLUSION 


The simplified gradient estimation algorithms, FSC and 


SLMS, have processing gain and convergence time similar to 


the classical LMS algorithm as shown in section 2.1. This 


Similarity of performance has been confirmed when all the 


filters which were compared have the same order, N° Thus, 


when one considers the implementation savings of the simpli- 


fied algorithms, the comparison favors the simplified ver- 


sions. 


—— 


Analytical comparison of the algorithms is 


possible using 


the results in sections 2.2, 2.3, and 2.4, which were developed 


for the adaptive line enhancer. 

A summary of these properties ‘ls presented 
and compared with the LMS algorithm eapertice 
C4]. 

Since E|x(k) |= k VR (0), it is clear from 
that the dynamic range of the FSC algorithm is 


because M and TC are not functions of Ry (0) - 


in Table 2.6-l1 


taken from 


Table 2.6-L 
the best, 


The LMS 


algorithm has the poorest dynamic range, since M and TC 


depend on Ry (9) - The SLMS algorithm is in the middle 


Since M and TC depend on the square root of Ry 


2.6-l1 presents the dynamic range properties of 


(0). Fig. 


these 


algorithms. Finally a sistematic approach to efficient 


adaptive filter design has been outlined. 
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Table 2.6-1 


Summary of Algorithms Properties 
Derived for ALE Example 


Misadjustment-M | Convergence Time-TC | Stability Limit 


1nl10 
Mon MUD 


Cu NR, (0) IPF 1/LN LR, (0)] 


(see note 2) (Eq. 2.3-12, (Eq. 2.2-20) 
see note 1) 


ab ven MO 


2(E]x(k) |)? 
(Eq. 2.431) 





u NLR, (0) a 
CaeTxtRyT PF yu, WN =| x(k) | 2/ON, E|x(k) |] 
(Eq. 2.4-30) 0 1.4) (Eq. 2.2-17) 
R__ (0) 
a in 
PF = 7 
min 


Notes: 


(1) A new LMS convergence time estimate, using (2.3=11) with 


Ooms = -955, is given by: 


Tc = 0-555 | = o} 
u(E| x(k) r YN. HRY, (9) WN 


(2) The LMS relationship taken from [4] with modification 
for the ALE example. This modification is similar to 


the derivation of (2.4-30). 


Wz 











A. PG YS INPUT POWER 





8 19 28 38 48 50 6e 78 88 93 1289 
RXX (8) 


B. TC US INPUT POWER 
868 
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@ 1 2 3 4 S 6 7 8 9 10 1 f@ 13 


RXX(@) 
LEGEND 3 
—v LAS 
veneces FSC 
---- SLRS 
Pag ceo 


Dynamic range of the algorithms Ni=10, His = ~0025 
Haga = Horus = .005, ALE simulation results average of 


£00 runs. 
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IIIT. RANDOM SEARCH IIR ADAPTIVE FILTERS | 


3.1 IIR PERFORMANCE SURFACE 

The performance surface of the IIR filter is much more 
complicated than the FIR performance surface because of the 
feedback of previous output values used to form the present 
output. 


Consider the filter: 
N71 Ny 


y(k) = = a,x(k-i) + = b. y(k-i) (3.1-1) 
0) fol 


and the error € , (k) = y(k) - s(k) . The MSE performance 


surface 1S given by: 


N -l 
2 _ . 
J({a.}, {b, }) = Ele _(k)] = Ee fe a, x(k-i) 
N 
b 2 
i oe Diy (ko )e r= s(k) J™} (3.1-2) 
eo 


In (3.1-2) k is large enough for operation of the filter to 
be in steady-state. 


Manipulating (3.1-2) we get: 
Nw -l N_-1l 
a a 
J(ia}, {b. }) = e 2 a. a, Ree $73) (3.1-3) 


+ 2 bs bs Roy $473) + Ri, (0) -2 2 


De Ree 
1 j=l y=. J SY J 
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where the correlation functions are: 


Ry (m) = E[x(k) x(k-m) J 

vy ™) = Ely(k) y(k-m) J 
R.,(™) = ELs(k) x(k-m) J 
Bay = E(s(k) y(k-m) ] 
Rey (m) = ELx(k) y(k-m) J] 
Rg (9) = E[s(k) s(k) ] 


Equation (3.1-3) apoears to be quadratic in the weights, but 
actually Roy (ms Ry (m) and Se also depend on the weights. 
The dependence of the performance surface (3.1-3) on the 
weights is of high order, and the surface has several minima, 
only one of which is the global minimum. To demonstrate the 
complexity of this performance surface consider the simple 


filter: 
y(k) = ax(k) + by(k-1) (3.1-4) 


One can recursively insert successive expresSions for y(k-i). 


enus : 
y(k) = ax(k) + blax(k-1) + b[ax(k-2) + bfax(k-3) + «---- 


(321-5) 


TS 








or in a compact form: 


k , 
y(k) =a f b” x(k-i) Soto) 


1=0 


J(a,b) = E{Ly(k)-s(k) 17} 





ko: . 
5 bb? ELx(k-i)x(k-j) J 


k : 
+ E[s*(k) 1] - 2a 5 b Elx(k-i) s (k)] 
i=0 
k i, 
were bias ser 
= 5 J a b R_(jrd) + R,, (0) 
ey 
— 2 2 -7 
Za se b Ro, (2) (So —7 ) 


Since R,,,. (™) and Ro, (m) do not depend on the values of the 

‘ weights a,b the degree of the performance surface given by 
(3.1-7) is already 2k for-b and quadratic ina. When the 
filter operates for a long time, k > ~ and (3.1-7) is an 
infinite sum and an infinite degree polynomial. 

Elliott, Jacklin and Stearns, [25], presents an expres- 
sion for the performance surface which is derived for the 
general case, with N. forward weights and Ny backwards weights, 
this expression is similar to (3.1-7). 

The general case also has quadratic depencence on the a's 
and infinite polynomial dependence on the b‘s. Thus the 
use of gradient search methods to optimize multimodal 


performance surface can be expected to result ina steady 
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state around one of the minima point which is not necessarily 
the desired global minimum. The steady-state tee point 
depends upon the initialization of the adaptive filter. This 
behavior has been demonstrated for the Stearns' algorithm 


by Parikh and Ahmed, [13] 


77 








3.2 THE RANDOM SEARCH CONCEPT 

A random search method consists of evaluation of esti- 
mates of the performance surface at discrete sets of the 
filter weights. After evaluation of these performance surface 
estimates, a comparison is made and a minimum point is 
selected. 

The method of selection of filter weights for whitch the 
performance surface estimate is to be evaluated, is very 
important. In order to have a useful adaptation scheme for 
non-stationary input signals a continuous search method is 
needed, in contrast to possible two phase method that has a 
global search phase and then fine tuning. From the several 
methods in the literature [19, 20, 21, 23, 30, 31, 33, 36] 
the needed continuity of operation is provided by the moving 
center method. 

The center is the point in the parameter space iw} with 
the lowest estimate of the performance function among the 
points that have been tested so far. The set of filter 
weights (or in general, system parameters) to be tested, at 


the £th random search interval, Wily, is given by: 


= W. + ug EOr alia (3.2-1) 


where: 

i is the parameter index 
Wile is the value of the center at the &%th random search 
interval, Ww, 


Q is its ith element 
# « 
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g 1s a number independently generated for each weight from 
a gaussian random number generator, with zero mean and 
unity variance 

u>O determines the range covered by one step of the search 

and is taken to be the same for all weights which need 


not be the case 


~ 


{w. } 
aM 


Q is a set of randomly selected parameter values 


around the center point tw, } during the 2th random 


ae 
search evaluation interval. Ww, Q 
td 


The test point (the set {Wi te) is tested; that is the 


~ 


is its ith element. 


1s estimated as J and 


| ed 


compared to the current center estimated value, J 


value of the performance surface, J g? 


Q° 


, a new point in parameter space is selected 


wn 


Edie di, 


using equation (3.2-1). If Jy < Jy the test point corresponds 


to a lower performance surface value estimate, and the center 


~ 


moves to a new location; that is we set W. = W. for all 
L,2£+1 ae 


i. Now (3.2-1) is used again to evaluate another point to be 
tested. 

Fig. 3.2-l1 presents a two parameter example of a moving 
center random search process. It should be noted that a, and 


are only an estimate of the performance surface points J, 


“ee 


Jy ; 


and J because the latter properly involves averaging over an 
infinite ensemble. 

In order to use the random search method in adaptive 
filters we need to specify the performance function and to 


define some estimate of that function. Since we are comparing 


the performance function and not evaluating it's gradient one 
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can select a complex non-analytic performance function. This 
possibility might be used to great advantage. However, in 
the following discussion we use the standard criteria of 
Minimum mean squared error. 

To evaluate an estimate to the performance surface we 
use two filters in parallel. One uses a set of weights that 
are the current center, so that this filter produces the 
output, y(k). The second set of filter weights correspond 
to a test point in the weight space. The filter output at 
the test point, On is used only during the adaptation 
process. 

The performance surface estimates are a time average, which 
is the only reasonable estimate of the ensemble average that 


we can calculate on line, and are given by: 


Jp = 2 Cy(k-j) - x(k-5) 7" (3. 2=2) 
J=0 

c oR. ; 

. 2, [y(k-j) - xr(k-3)] (62-3) 


where R is the number of input samples used to estimate the 
performance function for a given random search interval. 
We have two types of iterations. First filter iterations 
which process each new input sample with a fix set of filter 
Bweights and produce the outputs y(k) and ey The second 
type of iteration involves the random search selection of a 


new set of filter parameters which occurs after R filter 
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iterations. For each random search interval a new set of 
parameters is selected and tested. Fig. 3.2-2 presents the 


relationship of filter iterations to the random search in- 


terval. 


Fig. 3.2-3 presents a flow chart of the basic random 


search adaptive filter algorithm. 
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3.3 OPERATION OF THE RANDOM SEARCH ALGORITHM 
We now use the random search algorithm presented in 
Fig. 3.2-3 to implement an adaptive IIR filter. The filter 
operation is given by: 
N -l Ny 


a 
y(k) = L a, (k) X(k~1) + 


b, (k)y (k-1) (s5— 0) 
i=0 


1=1 
The weights {a, (k)} and {b, (k)} are functions of time and 
their variation is controlled by the algorithm of Fig. 3.2-3 
and equation (3.2-1). 
Details of the simulation are given in Appendix A. 
Fig. 3.3-l1 presents the operation of a random search IIR 
filter with N=3, Ny=4 H=-Ol, Uppal. 

We now discuss these results starting with some basic 
filtering considerations. 

The poles and zeros of a filter should be located sothat the 
desired spectral components pass through the filter and the 
unwanted components are rejected. 

For an adaptive filter we also need to match the filter 
output amplitude to the reference signal amplitude, that is 
there is a gain factor which must be adjusted accurately in 
the adavtive filter. 

The filter (3.1-1) has the following transfer function 


zZ 


T 

(Z—c) 

H(z) = are — = (3.3-2) 
nm? (Z-p,) 


1 
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where: 

q, are the filter zeros. There are M. zeros. 

p, are the filter poles. There are M. poles. 
The parameter ao controls the gain of the filter. For the 
adaptive filter we can't use the concept of transfer func- 
tion because the filter weights are time varying. However 
we Can consider an average steady state transfer function 
with weights that are the mean of the time varying weights. 
Thus an is required to match the filter maximum output magni- 
:tude to the reference amplitude.- En order to enhance a desired 
spectral component the filter should have a pole (or poles) 
near the spectral component, close to the unit circle. The 
effect of this pole on a signal at the same frequency would 
pe to multiply its amplitude by a gain factor of (1/1-p) 
where 9 is the pole's magnitude. Thus the output of the 
filter at the pole frequency is given by ao/ (1-09) times the 
magnitude of the input signal multiplied by a factor which 
depends upon the location of: the other poles and zeros. If 
this output magnitude is to be equal to the reference signal 
amplitude, a /(1-p) must have a specific accurate value 
because all the other factors that determine the output 
amplitude (namely the location of the other poles and the zeros) 
have only one optimal value, and thus for the steady-state 
near optimal filter are fixed, 

For good selectivity p is only a little smaller than 
Binity, and a, /(1-0) is the ratio of two very small numbers. 


It is difficult to achieve accuracy for this ratio with 
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random search adaptation on the forward weights (the a's) 
and the backwards weights (the b's or the poles). 
As a result of the mismatch of filter output and refer- 


ence signal amplitudes, decision mistakes occur when com- 


wn 
w~ ~ 


paring the performance function estimates, Jy > Jy with 
the result that a bad set of weights is sometimes chosen. 
These decision mistakes cause slower -convergence and smaller 
value of the steady-state o-(which means lower processing 
gain), and possible instability. This type of behavior was 
experienced in our simulation. One solution that we tried 
was to use smaller variance for the search on the feed- 
forward weights. This approach turned out to be inferior 

to a new approach (which is presented in the next section) 


based on the use of gradient search on the feedforward 


weights and random search on the feedback weights. 


88 








3.4 RANDOM AND GRADIENT SEARCH (RGS) 

The fact that the IIR filter's performance surface is 
a quadratic function of the feedforward weights, as dis- 
cussed in section 3.1, means that for non-varying feedback 
weights the performance surface with respect to the a's is 
unimodal. A unimodal surface may be handled best using a 
gradient method, and it is possible to achieve any desired 
accuracy to overcome the problem of the purely random 
search scheme discussed in section 3.3. 

Widrow and McCool [5], have compared a random search 
technique for a FIR filter with the LMS algorithm. The 
random search technique used was tailored to the unimodal 
Situation and, nonetheiess, resulted in inferior perfor- 
INMance compared with the LMS steepest decent gradient search. 

The question is how to make the feedback weights con- 
verge first, so that the feedforward weights would then 
converge to the global minimum. 

The cascaded arrangement, as shown in Fig. 3.4-1 is 
suggested. The all pole section comes first, and is 
adaptively controlled by a random search algorithm. A 
second all zeros section is then adaptively controlled by a 
gradient algorithm to produce, with suitable values of the 
adaptation gains, the desired effect of pole convergence 
followed by zero convergence, 

The optimal values of the adaptation gains is a com- 
promise of two contradicting considerations. The first factor 


is the requirement that the poles converge faster, and calls 
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for low Wo: The second factor is a desire to maintain enough 
randomness in the weight adaptation process so that we have 

a reasonable probability of transition from a local minimum 
zone to the global minimum zone, The effect of the later 
factor depends upon the specific shape of the performance 
surface. 

The internal signal $(k) has, at least during the pole 
convergence, non-stationary characteristics. In particular, 
the magnitude variations of $¢(k) are important to the 
adaptation cperation in the all zero section. As discussed 
in section 2.6 the FSC algorithm has no dynamic range limita- 
tions so that it is ideally suited for the RGS algorithm. 

The RGS algorithm is presented in three ways: Fig. 3.4-2 
presents its flow diagram, Fig. 3.4~-3 presents its block 
diagram, and Appendix B is a FORTRAN realization of the RGS 
algorithm. Fig. 3.4~-4 presents typical operation of this 
algorithm and shows the convergence and steady~state operation 
of the filter. Some further analysis and more simulation 


results are included in the next section. 
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3.5 CONVERGENCE OF THE RGS IIR FILTER 
Convergence of the RGS IIR filter is composed of two 

processes: the random search on the feedback weights, and 
the gradient search on the feedforward weights. These 
processes are coupled through the cascade structure of the 
filter, the common error exoression and the dependency of 
the feedforward weights on the poles magnitude, as dis- 
cussed in section 3.2. In order to analyze this situation 
we first assume independent operation and analyze each of 
the sections separately. We then combine the convergence 
time estimates with a correction factor to account for the 
fact that both processes converge simultanously. 

Consider first the analysis of the random search algor- 
ithm used in the RGS IIR filter in a general environment. 
In the simple case of single parameter, W, define the 


convergence zone: 


-WwW|< Aw (3.5-1) 


W, is the parameter, W, at the &£th random search 


ue 
* 
interval. W is the optimal value of W (the global mini- 
mum of the performance surface) 
* 
AwWw>0 is the limits of the convergence zone around W . 


A test point is selected by: 


ted 


Wea=W, tug (225-2) 
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where: 


~e 


We is the test point in the random search's £th interval. 
H>oO 1S. convergence control parameter adaptation gain. 
g 1s a number generated by N(1,0) random number generator. 


given We 1S¢ 


The probability density function of Wy 


(Wy 
exp[- 
V2TU 2u 


2 
Wy) 





“PAW, /W,) = J (3.5=3) 


and the probability of selecting a test point in the conver- 


gence zone is: 


* 
= = Cw & W +AW < e 
ee eel Ww | os AW/W = f P(W,/W, )aW, (3.5-4) 
W -AW 


Fig. 3.5-l1 illustrates the situation and the probabilities 


defined above. 


mrs 


Selecting a correct value for W, 1S not enough. After the 


ME 


testing of this point we need a correct decision that the 


tested point is better than W Thus we have the estimates 


Q° 
J and Jos and the selection of the correct weight depends 


upon their comparison. The probability of a correct decision 


~~ 


depends upon the values of Jy and Jos 


estimation parameters, mainly R. To simplify the analysis we 


their difference, and the 


define Pap (2) as the probability of a correct decision given 


m~. 


W,, averaged over all possible values of Wy: 


Q! 
We can write the probability of convergence to the con- 


vergence zone (3.5-1), 941 at the (2+1) random search interval, 


given W as: 


CY 
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p(w, /W,), JW) 
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The Selection Of A New Test Point 


oa, 





selection of W, in COLEECE decision, a 
Q+1 r- the convergence zone ronamely W,,,=W) 


The probability, Qos that the process does not converge in 


the first &£ iterations is given by: 


no convergence no conv. 


5 9 
as in the ls 
Qe F. Lrg. interval ; Pas we: inter. 
nmorconvy. inh Q 
: Se Leth inter. J = z [i-¢, J (3.976) 


The probability, Poy that the process does converge in the 


first % iterations is given by: 


(3.5-7) 
1 


So far we have discussed the single parameter case. It is 
now convenient to introduce the general case, namely M 


parameters. We can define the multidimensional convergence 


zone as given by: | 


(3.5-8) 
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where: 
Wo is a vector of M parameters, Wale 
W is a vector of optimal values, W, a eB a eee 
AW is a vector of deviations from the optimal values of 
the parameters defining the convergence zone. 


| | || is a norm defined on the parameter space. 


The multidimensional version of (3.5-2) is given by: 


* = W, + uG (3.5-9) 


where: 

G is a vector of M independent random numbers each of . 
which is N(1,0). 

Because of the independence of the parameters a multi- 


dimensional version of (3.5-4) is given by: 
MoM 
vo = TP, . (3.5-10) 


where P. is the single parameter probability given by 


ee 
B.5-4. 
The probability of convergence to the convergence zone at 


the 2+1 iteration given Wo is given by: 


P__ (2) (3.5-11) 
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_ 


pequations (3.5-6) and (3.5-7) remain the same for the multi- 
parameter case. An essential property of any optimization 
algorithm is it's ability to converge to the optimum. We 
Will now prove that the random search algorithm used in the 
RGS filter, converges to the convergence zone defined in 
(3.5-8). To observe this point we examine equation (3.5-7). 


Since (1-6) is a number always less than 1, the multiplication 


Q 
Il (1-6, ) becomes smaller as 2 increases. Thus 
i=l 
Rk 
P =lim P, = l-lim {fT [(1-6:J]} = 1-0 =1 (3.5-12) 
roe) Q : ab 
Q 00 9, 00 5 ke — ih 
P. = 1 means that after enough time the process converger to 


the convergence zone of the global minimum; that is, conver- 
gence with probability 1. Equation (3.5-12) does not provide 
quantitative information, namely an estimate of the convergence 
time. This problem is treated later in this section. 

Fig. 3.5-2 presents the results of a parameter identifica- 
tion experiment, the details of which are presented in Appendix 
A. This example was taken from reference [11] where it was 
used to demonstrate how Feintuch's algorithm converges to a 
point on the performance surface which is not a minimum. Ref. 
[13] uses the same example to demonstrate how Stearns' algori- 
thm converges to either a local or global minima depending 
upon the initialization point. Our results show that the RGS 


IIR filter converges to the global minimum even when started 
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in a local minimum point. Fig. 3.5-2 presents two experiments 
with typical convergence. 

The adaptation of a random search process has two types 
of parameter changes, or steps: 

- zone transitions - where after the movement, the 

center is near a new minimum 
- small steps - where after the movement, the center is 
near the same minimum. 

Fig. 3.5-3 illustrates the two step types. In signal filtering 
it turns out that the performance surface values at the local 
Minima are typically much higher than at the global minimun, 
as shown in Fig. 3.5-4. This difference in the performance 
surface value means that the random search process, when 
comparing values of the performance surface estimations, 
is insensitive to the local minima. In terms of step types, 
we neglect the analytically complex zone transition steps 
and analyze the situation typical of signal processing 
applications, assuming convergence with small steps only. 

We can define, for the general case of equation (3.5-9), 
the average step size, S__, as: 


av. 


ey = ELWs p41 = Wi 2 (S35-13) 


where: Wee) 


respectively. 


and We are the ith component of Woad and 
wy 
Using (3.5-2) as the equation for each component of (3.5-9) 


we have a 
- W. = ug | (3.5-14) 
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Since in small step convergence the direction of convergence 


is always in the same direction towards Ww, on the average 
only half of the test point are accepted, that is those 
where ug>0 (or those with ug<0). This assumes no decision 
errors when comparing two points. Using the above reasoning, 


from (3.5-13) and the definition of g=N(1,0) we get: 


2 u 
97/2, 2 (35-15) 
: dg 27 


oo co 

= Z ug p(g) dg = u i 
The above analysis ignores the coupling of the M parameters 
through the common error expression. At any interval one 
of the parameters is dominant, that is it contributes to the 
error term more than the others. As a result the correction 
of the value of this parameter dominates even though the 

-changes in the values of the other parameters may be in the 
wrong direction. 

Practically the value of a parameter can be expected to 
jitter around some value until this parameter becomes the 
dominant one. Then it's value would be corrected (and other 
parameters would jitter). If all the parameters were dominant 
for equal portions of the process the average step size would 
be 1/M of that given by (3.5-15), but parameters with larger 
numerical value get more attention, and the reduction in the 
average step size is given by: 


(3.5-16) 


= 


ae 
av yt x 


D 
Seg is the average step size for the dominant, large 
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valued, parameter. 0 <0 <1 is unknown factor. 
Assuming that there is a correct decision, the mean nun- 
ber of random search intervals needed for convergence is 


W ax/ Si? where Wee is the value of the largest parameter; 


W 
Meee) Mase: 7 
Te = by a R (3.5-17) 
av 
t 
where: a 


1 1s a proportionality constant that depends upon the 
exact definition of TC (103, a Of ANLt1lal- error, 
etc.) 

- TC in (3.5-17) is given in filter iterations. We can combine 


(3.5-16) and (3.5-17) and include /27 in a, to yield: 
W ane 
TO=a, °-— = -MO*R (3.5-18) 


Equation (3.5-18) provides a convergence time estimation 

for a general case of random search operation with the above 
assumptions. Let us turn now to the RGS IIR filter and get 
a specific expression for its' convergence time estimate. 
For the RGS IIR filter case equation (3.5-18), in terms of 


the algorithm parameters as defined in section 3.3, becomes 





N, OR (3.5-19) 


To get an estimate to Deas we consider the transfer function 


of the filter given by equaiton (3.1-1): 
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Ny (SS 410) 


where: 
N. is the number of 2-order sections of the filter, N= 
NL /2 : 
es 1s the magnitude of the jth pole 
ba = cos (2nf/fs) 
f - pole frequency, fs - the sampling frequency. 


pas 


mor a stable filter fas and the largest possible Dax 


given by additions of the terms tabs which is the coefficient 


of z 7 when the multiplication of (3.5-20) is expanded 


2= 2N . = 2(N,/2) = Ny (3.5-21) 


cee Bry as) 


a, =a, + 1 is expected to be in the range 1<a4<2. 
Equation (3.5-22) estimates the convergence time of the feed- 
back portion of the RGS IIR filter. To estimate the conver- 
gence time of the feedforward portion we can use the FSC 


relation (2.3-1). As discussed in section 3.3, the largest 
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value of a feedforward weight should be proportional to (l-p) 


a se = a, (1-p) (3.5-23) 
where op is the pole's magnitude. In addition to the dis- 
Cussion of section 3.3 which relates the gain factor ao to the 
dominant pole magnitude pop, it is also noted that the adapta- 
tion gain is also related to the pole magnitude because the 
smaller the adaptation gain the finer the control of the 

ratio ao/(1-p) and the closer the pole magnitude can be 
adjusted to the unit circle. Thus we write the following 
relationship: 


Os 
(l-p) = Ayu, = (3.5-24) 


where Oy and a. are proportionality constants. Fig. 3.5-5 


5 
shows the transient and steady state values of the pole 
magnitude for several values of ts The non-linear 
relationship between p and Ho, as suggested by (3.5-24) seems 
to be reasonable. 

Combining (2.3-1), (2.3-4), (3.5-23), (3.5-24) and value 


of a ,=1/2 in (2.3-1) we get: 


/N_ (3.5-25) 


We now determine the convergence time for the RGS IIR filter 


by adding (3.5-25) and (3.5-22): 
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Gir Gor Ge, Go are experimentally evaluated constants. The 
values of ay and a- can be adjusted to compensate for the 
Simulataneous convergence of the all pole and all zero sections 
of the RGS IIR filter. 

A special difficulty encountered is to confirm the de- 
pendence of the random search convergence time upon the 
number of weights, Nye in the random search process. A 
Seance in the number of feedback weights in the RGS IIR 
filter (with or without changing the input signal) causes 
Major changes in the nature of the problem to be solved. For 
example if there are more poles than necessary, only one of 
them needs to converge, and the others are cancelled by the 
zeros. Any experiment in which the number of poles is varied, 
(with or without changing the signal) will combine the effects 
of changes in the nature of the problems to be solved, the 
effect of any changes in the input signal statistics, as well 
as the effect of more parameters upon the random search con- 
vergence time. A simpler approach is to construct a random 
search FIR filter and to use this filter to verify the analysis 
of the random search process and the dependence of its con- 
vergence upon the number of weights. 


We start with relation (3.5-18) using FIR notation. Thus 


a 
femme 22 2 2 RP | (oni5a07) 
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From Treichler [37] we have: 


SNR 


“max  T¥.5N_, SNR rea. 
where SNR is the input signal to noise ratio. 
Inserting (3.5-28) to (3.5-27) gives: 
Cl 
SNR + N, at) 
“<= 91 (IFTEN_ SNR) 1, (a ). 


Fig. 3.5-6 presents a comparison of simulation results with 
convergence estimate (3.5-29) for several values of Noe with OS 
and o, experimentally determined as a, = -4242, a4 = le 
These results verify the ability of equation (3.5-18) to 
estimate the effect of the number of parameters on the con- 
vergence time of a random search filter. 

An important assumption used in the convergence analysis 
of the RGS IIR filter was that the random search process does 


~~ 


not make mistakes in the comparison of J >< Ty. In order 


R 
to define conditions for filter operation with no decision 
mistakes we investigate the effects of the random search 
interval, R,.. on the RGS filter performance. Fig. 3.5-7 
presents operation with decision mistakes. This situation is 
typical in operation with relatively small random search 
interval. In the example of Fig. 3.5-7 we used R = 100. 

For larger values of the random search interval, R, 


we have slower convergence as given by equation (3.5-26) and 


illustrated by the pole convergence of Fig. 3.5-8, Big S.o-c 
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also shows that when the random search interval is too large 
(R = 800 in the example of Fig. 3.5-8) the pole magnitude 
is smaller and the resulting signal processing gain is lower. 
This unwanted effect is caused by decision mistakes that occur 
because with a long random search interval the feedforward. 
weights provides better match for the weights in the center 
filter than for the tested filter for which the feedback 
feedforward weights are only copied (Fig. 3.4-3). This effect, 
of smaller pole magnitude, for long random search interval 
depends on the convergence of the all zero section and hence 
“upon the value of Uo: 

The above discussion suggests that there is an optimal 
value for the random search interval, which depends upon the 
value of Ho: From the results obtained in our experiments 


eam dene 


it seems that for He in the range of 30x10. 
optimal values of R are in the range 300 to 500 iterations. 

To verify the convergence time estimate of equation (3.5-26) 
we present experiments with several values of ue (Fig. 3.9-9) 
and several values of Up (Fig. 3.5-10); all of them with R=500 
to assure that the operation is practically free from decision 
mistakes. 

The effects of He as discussed above are clear in Fig. 
B.5-9: 

(1) The steady-state value of the pole magnitude is 

closer to 1 for smaller Wo: 


(2) The convergence rate of the feedforward weight is 


proportional to Ho: 
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(3) The overall effect of the convergence and steady- 
state value of the output MSE is a combination of 
(1) and (2). 
The effects of Hp, as discussed above are clear in Fig. 
Be5-10. 
(1) The convergence of the pole is proportional to Hy - 
(2) The overall effect in the convergence of the output 
MSE is mainly the convergence rate. 
It is interesting to note in Fig. 3.5-10 that the convergence 
rate of the feedforward weight is equal for all the values 
oq Hy: 

Table 3.5-l1 compares the convergence time measured in 
the experiments presented in Fig. 3.5-9 and Fig. 3.5-10 to 
the estimation given by (325226) with experimentally deter- 
mined proportionality constants. The modified estimation 


formula for N =2 is: 


me _ 


TC = AVR A 1 OS alta + 3.8 R/U, e530) 


Table 3.5-1 shows good agreement between experimental 
measurements of RGS filter convergence time and the esti- 
mations of (3.5-30). This agreement verifies the analysis 


of the RGS IIR filter convergence properties. 
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Table 3.5-1 
RGS Convergence Time Measurement and Estimation 


(Results are an average of 32 runs with No=3, N,=2) 
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45,334 
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Fig. 3.5-10 


64,372 


66,000 


Fig. 3.5-10* 


100,000 4102,448 





NOTE 


* Convergence time for #5 cannot be measured from Fig. 3.5-10 


because only part of this experiment data is plotted here. 
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3.6 APRIORI STRUCTURED ADAPTIVE FILTERS (ASAP) 

Motivated by the possibility to reduce the number of 
variables under random search adaptation, we now investigate 
the relationship of the filter's weights to a smaller set of 
variables. In some cases the structure of the optimal filter 
is known and only a few parameters are unknown and need to be 
evaluated. In other cases we might accept a sub-optimal 
simpler solution, namely an optimal filter with structural 
constraint. For the filter (3.1-1) we might have a smaller 


set of parameters, W, such that: 


a, = £,(W)°  i=0,1,...,N_-1 (3.6-1) 


i i= ‘. is (3.6-2) 


where: £,(:) and g, (+) are functions that connect each 
filter weight to the parameter vector W. 

Since we have a good solution to the feedforward weight 
adaptation, it is acceptable to use the same combined random 
and gradient search method that was used for the RGS IIR 
filter, for the proposed ASAF filter as presented in Fig. 
3.6-1. 

We will continue the discussion by considering the speci- 
fic case of a pole close to the unit circle with adaptation 
of it's frequency only, see Fig. 3.6-2. 


This type of adaptive: filter is useful for the ALE 
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application, or when several such sections are cascaded to 
enhance a multiple sine wave signal. 

Fig. 3.6-3 presents a typical operation of an Apriori 
Structured pole, ASPOL IIR adaptive filter with pole magnitude 
o = .99. Simulation details are presented in Appendix A. 

It is clear from Fig. 3.6-3 that this filter has high 
processing gain with relatively short convergence time. 


These advantages are also shown in Fig. 1.5-l. 
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3.7 CONCLUSION 

An IIR filter has, in many cases, great advantages over 
a FIR filter because of the efficiency associated with the use 
of recursion and the existence of poles in the transfer func- 
tion. However, realization of an adaptive IIR filter is a 
difficult task because of the multimodal nature of the IIR 
filter's performance surface, as well as the stability 
problem and the complexity of the performance surface 
gradient expression. 

The proposed Random and Gradient Search (RGS) algorithm 
Overcomes the complexity of the multimodal performance sur- 
face and converges to the global optimum with probability l. 
This convergence is guaranteed for sufficiently large time. 
For the important Special case of large difference in the 
value of the performance function between the global minimum 
and local minima, an estimation for the average convergence 
time has been derived and verified by simulation results. 


The convergence time estimate is given by: 
- vs 
2 


oo 5 
' TC = ay YN +a, ° - R (357-1) 
ba a 1 Hy 





where G-, Go, A, a) are experimentally evaluated constants. 
Stable operation of the RGS IIR filter was demonstrated 
in many hours of computer simulation without overflow 
problems. This stability is attributed to the detection of 
excessive MSE at the tested points, and the fact that the 
algorithm discards such points before overflow, which is 


Causes by unstable filter weights, can be developed. Thus 
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the RGS is a practical candidate to realize an adaptive IIR 
eilter. 

The Apriori Structured Adaptive Filter (ASAF) uses the 
random search method and additional structure information 
to improve adaptive IIR filter performance. The moving 
pole example, for instance, is guaranteed to be stable and 


has fewer parameters in the adaptation process. 
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IV. SUMMARY 


Two approaches to efficient adaptive filtering have been 
investigated; a FIR filter with simplified gradient estima- 
tion methods, and IIR filters with a combined random search 
and gradient adaptation scheme. 

Two simplified algorithms, the Fixed Step Correction (FSC) 
and the Simplified LMS (SLMS), are derived and compared to the 
Classical LMS algorithm for the FIR filter. The comparison 
includes analysis of filter SEeOaE Racer and extensive simu- 
lation results are presented to verify the analysis. 

Because the adaptive filter properties depend upon the 
Statistics of the input signal, the desired signal, and the 
reference signal, when analyzing the operation of an adaptive 
filter one must assume some statistics for the above signals. 
Hence, the analysis is valid for a specific case or a class 
of cases. 

Thus algorithm comparisons and the adaptive filter pro- 
perties analysis has been carried out here for the applica- 
tion of the adaptive line enhancement (ALE). The analysis 
includes convergence time estimate, steady-state misadjustment, 
and filter processing gain. Estimates to these properties 
have been derived and verified by simulation results which 
compare the three algorithms (LMS, FSC, SLMS). The conclu- 


sion of the comparison is: that for equal filter order the LMS 
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algorithm is somewhat better. However when one considers an 
equal complexity, which allows the use of a higher degree 
filter for the simplified gradient estimations, the result is 
that the FSC and the SLMS are better than the classical LMS. 
The IIR filter offers, in many cases, computational 
Savings over a FIR filter. However, the IIR filter has 
a multimodal performance surface and may be unstable. Be- 
cause of these two problems, and the complexity of the gra- 
dient expression, the algorithms which have been proposed for 
the IIR adaptive filter by Feintuch and Stearns do not provide 
a satisfactory solution. The Random and Gradient Search 
algorithm (RGS) proposed here has the ability to converge 
to the global minimum of the multimodal performance surface, 
and convergence with probability one is guaranteed for suf- 
ficiently large time. For the important class of cases 
characterized by a global minimum much lower than the local 
minimum, an average convergence time estimate has been de- 
rived and verified with simulation results. The use of 
structure information of the optimal solution when known 
allows the construction of an Apriori Structured Adaptive 
Filter (ASAF). This version of the RGS IIR filter optimizes 
a smaller set of parameters and is advantageous in some prac- 
tical applications. In summary this research has demonstrated 
that with the RGS scheme it is possible to realize an adaptive 
IIR filter which will operate properly and have a practical 


implementation. 
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At the end of this dissertation it is appropriate to 
indicate some of the subjects that call for further work: 
In the area of FIR adaptive filters with simplified gradient 
estimation, some topics are: 
- Effects of finite arithmetic. 
- The study of variations of convergence and steady- 
state behavior around the mean. 
- The dependence of convergence and steady-state behavior 
on input signal to noise ratio. 
- Operation with complicated signals. 
- Extension of the discrete algorithms developed here to 
analog systems, including adaptive antenna arrays. 
- Consideration of non-stationary input signals. 
- Applications. 
In the area of RGS IIR filters; topics include: 
- The study of random search decision error dependence 
upon filter parameters. 
- Analysis of possible processing gain dependence upon 
filter parameters. 
- The effects of operation with an inaccurate random 
number generator. 
- Possible configurations for apriori structured adaptive 
filters. 
- Operation with complicated signals. 
- Consideration of non-stationary input signals. 


- Applications 
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APPENDIX A 


SIMULATION 


A.1l THE SIMULATION METHOD 

Simulation was used to provide experimental data for 
comparison of adaptive filtering algorithms, and for veri- 
fication of Er ait formulas. The simulation program includes 
four basic functions: 

(1) Execution initialization: Signal parameters (fre- 
quency, Signal to noise ratio, power,..etc.) and filter 
parameters (filter type, number of weights, adaptation gain, 
~.-, etc.) are loaded interactively into the computer to con- 
trol the forthcoming execution. 

(2) Experiment configuration and signals generation: 

A signal generator subroutine, determined by the signal 
parameters loaded in the execution initialization, prepares 
sequences of 100 samples of input and reference signals to 

be processed by a filter subroutine. The relationship of the 
input and reference samples determines the adaptive filter 
simulation to be evaluated, i.e. ALE or parameter identifica- 
rion. 

(3) The filtering function: As controlled by the filter 
parameters loaded during the execution initialization, a 
filter subroutine is called upon to process the data in blocks 
of 100 samples. 

(4) Experiment data extraction and storage: The program 


stores the values of J (k) and 200 values (spread equally 
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over the time of the experiment) of up to six parameters, 

and at the end of the experiment the program evaluates the 
processing gain and the convergence time for that experiment. 
The foregoing information, together with the experiment 
parameters, are stored on disk files and are available for 
off-line use. 

A simplified flow diagram of the simulation program is 
presented in Fig. A.1l-1. It should be noted that although 
the FIR and IIR simulation programs have identical structure 
there are some differences as discussed in sections A.2 and 
A 3 . 

A data handling program is used to access the data files 
and present experimental results in tabular or graphic forms. 
The graphic option includes plots of variables as function 
of time and as functions of a filter or a signal parameters. 

The simulation was done on a PDP-11/50 minicomputer under 


RSX-11M multiuser operating system. 
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A.2 FIR SIMULATION PROGRAM 
The FIR simulation uses the Adaptive Line Enhancement 
(ALE) configuration as presented in Fig. A.2-1. The desired 


signal is: 
s(k) = v 2R_ (o)' cos wk (A.2-1) 


where Ro, (0) is the desired signal power. 


And: 
x(k) = s(k) + n(k) 


where n(k) 1S a white gaussian noise with variance Rin '?: 
The execution initialization controls the parameters Ri, (0), 
Ran’): and w. For each N. and signal statistics used, the 


program evaluates the optimal values of the mean squared 


* 
SPErOr, Sr and weights a, as follows: 
eA = lim J (k) (A.2-2) 
ea 2 
k > @ 
* « s 
a, = lim a, (k) ots i=0,1,...,N -1 (A.2-3) 
Ie 
k7> © 
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For 200 points spread equally over the time of the experiment ~ 
the program evaluates the performance function, J(k): 


apr) J nin + [A,-A R [A,-A 4 (A. 2-4) 


where 
A, is the vector of filter weights at time k. 
A" is the vector of optimal weights determined by (A.2-3). 
R is the input signal autocorrelation matrix. 
At the end of the experiment the program evaluates: 
(Ll) The steady-state MSE, ee as the average value of 
J(k) in the last 10% of the experiment. 
(2) Convergence time, TC, directly from the definition 
as the time required for the error (J(K)-To 0) to be 


reduced to 10% of its initial value. 


(3) Misadjustment, M: 
‘= — (A. 2-5) 


(4) Processing gain, PG: 


(0) 


R 
PG = 10 log [_——] (A. 2-6) 
ss 


The program includes filter subroutines that perform the ’LMS, 


| FSC, and the SLMS algorithms. 
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A.3 IIR SIMULATION PROGRAM 
Three configurations were used for IIR adaptive filters 
simulation: 
- ALE configuration as discussed in Section A.2. Dif- 
ferences from the FIR program are indicated later. 
- Parameter Identification example as shown in Fig. 
A.3-1. This example was used to demonstrate conver- 
gence to the global minimum of a multimodal performance 


surface. This example is taken from Johanson and 


Larimore [11] and was used also by Parikh and Ahmed [13]. 


- Stability test, as shown in Fig. A.3-2. This test was 
used with the optimal location for the pole near the 
unit circle to demonstrate the stability of Feintuch 
and the random search algorithms, and the lack of 
stability of the Stearns' algorithm. 

The ALE IIR experiments are similar to the FIR except for 
the following: 

(1) The optimal values of (A.2-2) and (A.2-3) are not 

used. 

(2) The MSE, J(k), 1s evaluated by: 


Nees 
av 
J{kj = ¢ 


J 


aay, 
5 Ey eo) eso (Koa) Ieee (as —40) 


where Nee is an averaging interval, the values used were 
between 100 and 500. Obviously J(k) is evaluated only once in 


each averaging interval. 
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(3) The misadjustment, M, of (A.2-5) is not used. 
The program includes filter subroutines that use the 
following IIR algorithms: 

- Feintuch 

- Stearns 


Random Search 


- RGS 

- Apriori Structured Pole. 
In order to present the algorithm details and as an example 
of a filter subroutine, the RGS subroutine is given in 


Appendix B. 
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APPENDIX B 


SUBROUTINE RGS 


= 
OOOO OOO OOOO OOOO OOO IK IO ACKIK 
C x 
| = RANDOM AND GRADIENT SEARCH SUBROUTINE x 
& x 
c X(100) INPUT DATA IN 100 ELEMENTS ARRAY x 
: C R€100) REFERENCE DATA * 
C Y(100) OUTPUT DATA x 
c AC + ) FEEDFORWARD WEIGHTS x 
S BC +« 2) FEEDBACK WEIGHTS x 
& x 
OOOO OIC ICI IOI KICK ACHCOK 
C 
e THE COMMON BLOCK 
INCLUDE ’AR.CMN’ 
DO 100 K=1%100 
C FEEDBACK SECTION PROCESSING,» 
e BT(.) IS THE TESTED FOINT. 
YN=XCK) 
YTN=X(K) 
DO 101 J=1»NB 
YN=YNtBC J) KXYBCJ) 
YTN=YTN+BT CJ KYT CU) 
101 CONTINUE 
& SHIFTING THE SIGNAL IN THE FILTER’S MEMORY 
BDO 102 J=1»NAR-1 
YBCNABR-J+1 =YBCNAB-J) 
YT CNAB-J4+1)=YTCNAB-J) 
) 102 CONTINUE 
YBC1L)=YN 
YTCL)=YTN 
C FEEDFORWARD SECTION PROCESSING 
Y¥(KRd=0.0 
ZT=0.0 
NO 103 J=1*NA 
YCRIA¥CRIFtACI) KYBCI) 
ZT=ZT+tACIJIKYTC I) 
103 CONTINUE 
B THE ERROR TERMS 
ER=Y(R)-RCK) 
ET=ZT-R(K) 
S FEEDFORWARD WEIGHTS’ ADAPTATION 
UO 104 J=1+*NA 
ACJ =ACJ)-GAXSIGN(1-.Or7ER) XSIGN(1.OrYBC(J)) 
104 CONTINUE 
c FERFORMENCE FUNCTION ESTIMATION 
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EC=ECtERXER 


ECT=ECTTETXET 
IFCECT.GT.TH) L=LL 


L=L+1 
IF(L.LE.LL) GO TO 111 
C COMPARISON AND RANDOM SEARCH DECISION MAKING 
L=0 
IFCECT.LT.EC) GO TO 200 
GO TO 201 
200 CONTINUE 
DO 203 J=1l»NB 
BC JO=BT (I) 
YBCJO=YT (I) 
203 CONTINUE 
201 CONT INUE 
C NEW TEST POINT SELECTION 


DO 204 J=1,NB 
BT (J) =B(J)+GBXGAUSS (0) 


204 CONT INUE 

NO 205 J=1»NAR 

YT(J=YBCJ) 

205 CONT INUE 

EC=0.0 

ECT=0.0 
so CONTINUE 
C : EXPERIMENT DATA EXTRACTION 


CALL AVERR(R) 
IFCIOUT.EQ.2) CALL WTPFR 
100 CONT INUE 
RETURN 
END 
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